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Introduction 

This  report  summarizes  the  accomplishments  of  a  multi-year  research  contract  entitled  “ Virtual 
Teleoperation  for  Unmanned  Aerial  Vehicles funded  via  direct  congressional  appropriation  to  Iowa  State 
University  and  administered  through  the  AFOSR/RL. 

The  initial  grant  (FA9550-05-1-0384)  was  funded  in  FY05  for  approximately  $2.8M,  for  the  period 
5/15/2005  through  5/14/2007.  In  FY06  it  was  supplemented  by  another  congressional  appropriation  for  an 
additional  $3.3M  and  the  end  date  was  extended  for  an  additional  two  years  to  5/14/2009.  In  FY07,  a  third 
and  final  congressional  appropriation  was  directed  for  the  project  totaling  $4.3M.  Rather  than 
supplementing  the  original  grant  (FA9550-05-1-0384)  a  new  one  was  initiated  (FA9550-07-1-0475)  with 
the  same  title  and  a  revised  statement  of  work.  Thus  the  total  budget  for  the  “Virtual  Teleoperation 
for  Unmanned  Aerial  Vehicles”  project  sums  to  $10. 4M  for  the  period  May  2005  through  May  2009.  In 
May  2009  a  no  cost  extension  of  the  project  was  granted  for  an  overall  end  date  of  May  2011. 

Summary 

The  focus  of  this  project  is  the  exploration  of  next-generation  command  and  control  interfaces  for  UAVs 
based  on  advanced  virtual  reality  and  other  emerging  interface  technologies.  Part  of  our  rationale 
for  requesting  such  a  substantial  budget  was  the  need  for  a  major  investment  in  our  infrastructure  to  enable 
the  research.  Although  the  research  infrastructure  at  Iowa  State  University's  Virtual  Reality  Applications 
Center  was  substantial  when  the  project  was  initiated  in  2005  it  was  inadequate  to  support  the  vision 
outlined  in  our  proposal.  For  example,  our  two  premier  immersive  interfaces,  the  fully  immersive  six-sided 
CAVE  called  the  “C6”  and  the  stereoscopic  display-enabled  Lee  Liu  Auditorium  were  both  developed  in 
1999  when  Howe  Hall  was  built.  Thus,  our  proposal  called  for  substantial  investment  in  enhancing  both 
of  these  leading  edge  devices.  Although  the  research  effort  began  immediately  and  ran  concurrently 
with  the  infrastructure  enhancement  projects,  the  majority  of  the  expenditures  during  the  first  two  years  of 
the  project  (2005-2007)  focused  on  enhancing  the  C6. 

The  C6  enhancement  was  a  truly  revolutionary  project  -  we  worked  closely  with  commercial 
vendors  including  Mechdyne  and  Sony  to  push  the  boundary  of  resolution  and  image  generation  far  beyond 
what  existed  before.  In  fact,  the  requirements  we  generated  during  this  process  enabled  Sony  to  expand  the 
market  for  their  new  SXRD  (LCOS  projection)  technology,  designed  originally  for  the  digital  cinema 
market,  into  the  visualization  and  simulation  market.  At  a  total  of  100M  pixels,  the  C6  is  now  the 
world’s  highest  resolution  fully  immersive  VR  display  system.  The  C6  enhancement  has  been  an 
unqualified  success,  not  only  enabling  the  research  we  originally  proposed,  but  also  many  new 
research  opportunities  for  our  colleagues  at  VRAC.  It  garnered  national  recognition  from  the  popular  press 
including  Wired  Magazine,  the  Discovery  Channel,  and  CNN.  It  also  opened  new  doors  within  the  DoD, 
including  new  research  relationships  such  as  our  activities  with  colleagues  at  Wright  Patterson  Air  Force 
base.  We  have  hosted  visits  from  colleagues  at  NASA  Ames,  and  have  visited  colleagues  atAFRL's 
Human  Effectiveness  Branch  in  Mesa  Arizona  to  discuss  high  resolution  displays.  In  addition,  in  December 
2008  Professor  Oliver  was  invited  to  brief  the  USAF  Operational  Based  Vision  Assessment  (OBVA) 
Integrated  Product  Team  on  our  success  in  developing  such  a  high-resolution  display  system  at  the  I/ITSEC 
Conference  in  Orlando,  Florida.  Since  its  opening  in  2007,  to  our  knowledge  only  one  other  equivalent 
system  has  emerged:  the  CORNEA,  based  at  King  Abdullah  University  of  Science  and  Technology 
(KAUST)  in  Saudi  Arabia  opened  in  2009.  Built  by  Mechdyne  as  well,  the  CORNEA  is  essentially  a 
replica  of  ISU’s  C6. 

While  pushing  the  boundary  as  we  did  paid  substantial  dividends,  it  took  much  longer  than  we  had  initially 
planned.  In  addition  to  enumerable  technical  challenges  faced  by  our  vendors  and  their  suppliers, 
we  encountered  more  pedestrian  but  unforeseen  challenges,  such  as  enhanced  power  and  HVAC 
requirements  for  the  C6,  that  set  the  schedule  back  repeatedly.  After  a  total  investment  of  approximately 
$5M,  the  C6  enhancement  was  declared  “completed”  in  May  2007,  although  ongoing  debugging  and  tuning 
with  Mechdyne  and  Nvidia  lingered  through  summer  2008  leading  to  a  final  incremental  enhancement  of 
96  new  graphics  cards  for  the  C6  image  generator  in  fall  2008. 
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In  2007  the  infrastructure  enhancement  emphasis  shifted  from  the  C6  to  the  Lee  Liu  Auditorium.  While  this 
project  was  of  substantially  smaller  scope  and  complexity,  and  we  had  learned  much  from  the  C6, 
it  nonetheless  presented  its  own  challenges.  In  order  to  accommodate  the  two  Sony  SXRD  projectors,  the 
control  room  of  the  auditorium  had  to  be  substantially  remodeled  which  required  another  engagement 
with  university  facilities  management  architects  and  their  procedures.  The  Lee  Liu  Auditorium  upgrade 
was  completed  in  April  2008  for  a  total  cost  of  about  $800K  making  it  the  world's  highest  resolution 
stereoscopic  auditorium  display. 

Throughout  these  infrastructure  enhancement  projects,  our  research  efforts  continued  to  grow  and  gain 
momentum.  The  project  supported  the  research  of  numerous  faculty  members  (with  particular  emphasis  on 
junior  faculty),  several  post-doctoral  researchers,  and  many  graduate  and  undergraduate  research  assistants. 
Project  personnel  worked  in  a  coordinated  fashion  on  complimentary  aspects  of  the  overall  vision. 

Research  results  are  documented  at  our  project  web  site:  http://www.vrac.iastate.edu/uav.  Research 
supported  by  the  grant  has  garnered  three  best  paper  awards  at  national  conferences,  enhancing 
our  visibility  and  competitiveness  for  related  grants.  Graduates  of  our  program  are  now  employed  by  DoD 
contractors  such  as  Boeing,  Lockheed  Martin,  and  Rockwell  Collins,  in  addition  to  many  others  in  the 
software  and  other  high  technology  industries.  Of  the  total  $10.M  budget,  approximately  $6M  has  been 
expended  in  the  research  infrastructure  described  above,  and  approximately  $4M  in  personnel. 

Personnel  Supported 

Through  the  six-year  duration  of  the  project,  the  following  personnel  were  supported  by  the  grant: 

Faculty: 

PI: 

•  James  Oliver,  Larry  and  Pam  Pithan  Professor  of  Mechanical  Engineering 
Co-PIs: 

•  Eliot  Winer,  Associate  Professor  of  Mechanical  Engineering 

•  Stephen  Gilbert,  Assistant  Professor  of  Industrial  and  Manufacturing  Systems  Engineering 

•  Derrick  Parkhurst,  Assistant  Professor  of  Psychology 

•  Alex  Stoytchev,  Assistant  Professor  of  Electrical  and  Computer  Engineering 

Senior  Personnel: 

•  Soon-Jo  Chung,  Assistant  Professor  of  Aerospace  Engineering 

•  Chris  Harding,  Assistant  Professor  of  Geologic  and  Atmospheric  Sciences 

•  Jonathan  Kelly,  Assistant  Professor  of  Psychology 

•  Nir  Keren,  Assistant  Professor  of  Agricultural  and  Biosystems  Engineering 

•  Rob  West,  Associate  Professor  of  Psychology 

•  Arun  Somani,  Anson  Marston  Distinguished  Professor  of  Electrical  and  Computer  Engineering 

Post-doctoral  Researchers: 

•  Vijay  Kilavarapu 

•  Eric  Foo 

•  Keehong  Seo 

Graduate  Students: 

•  22  PhD  Students 

•  28  MS  Students 

•  Numerous  undergraduate  research  assistants 
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Research  Accomplishments 

In  addition  to  the  infrastructure  enhancements  described  above,  the  following  research  contributions  are 
summarized  below: 

Virtual  Battlespace  -  is  a  flexible  VR  software  platform  to  support  our  research  in  for  virtual  teleoperation 
of  UAVs.  The  Virtual  Battlespace  integrates  information  about  tracks,  targets,  sensors  and  threats  into  an 
interactive  virtual  reality  environment  that  fuses  available  information  into  a  coherent  picture  that  can  be 
viewed  from  multiple  perspectives  and  scales.  Visualizing  engagements  in  this  way  is  useful  in  a  wide 
variety  of  contexts  including  historical  mission  review,  mission  planning,  pre-briefing,  post-briefing  and 
live  observation  of  engagements  in  progress. 

Virtual  Battlespace  immerses  users  in  a  virtual  environment  that  provides  them  with  greater  context  and 
awareness  of  the  units  under  their  control  as  well  as  the  overall  mission.  By  integrating  radar  tracks  and 
UAV  video  feeds,  the  virtual  world  can  provide  access  to  the  latest  real  time  battlefield  information.  The 
virtual  world  is  constructed  from  a  mix  of  a  priori  information  and  real  time  sensor  feeds  to  act  as  an 
organizing  context  for  the  operator,  resulting  in  a  mixed  reality  system,  in  which  real-world  video  and  radar 
tracks  augment  a  dynamic  virtual  world.  Using  real-world  data  to  augment  the  virtual  world  is  an  inversion 
of  the  more  typical  paradigm  of  augmented  and  mixed  reality  where  virtual  information  is  used  to  enhance 
real  world  data  and  imagery. 

To  facilitate  modular  enhancement,  Virtual  Battlespace  incorporates  a  command  architecture  that  abstracts 
message  passing  among  components  as  well  as  stream  input  so  that  any  arbitrary  input  stream  can  be 
accommodated  including  terrain  generators,  path  planners,  and  voice  recognition  input. 

Some  unique  features  of  the  Virtual  Battlespace  include: 

•  Ability  to  read  in  terrain  data  in  most  open  formats  such  as  DTED  and  GIS. 

•  Models  such  as  SAM  sites,  aerial  and  ground  vehicles,  and  other  graphical  objects  displayed  with 
varying  levels  of  detail  (LOD)  and  aggregation/disaggregation  glyphs  based  on  viewer  distance 
and  function. 

•  Ability  to  render  environment  for  stereographic  viewing  over  arbitrarily  large  cluster  computers 
and  display  channels 

•  Communication  server  implementation  to  listen  for  DIS/HLA  updates 

•  Integration  of  spatial  sound  in  an  immersive  environment  to  facilitate  interface  experimentation 

•  Semi-automated  UAV  path  planner  incorporating  constraints  for  reconnaissance,  threat  avoidance, 
and  terrain  features 

•  OneSAF-based  integration  of  multiple  simulators  for  multimodal  interaction 

•  Generally  abstracted  controller  hardware  and  graphical  interfaces  facilitating  deployment  on  a 
variety  of  VR  platform  architectures 

Sparsh  UI  -  To  facilitate  multi-modal  interaction  via  multi-touch  surface-based  interaction,  a  research 
team  led  by  Professor  Stephen  Gilbert  developed  and  released  Sparsh  UI  (http://code. google. com/p/sparsh- 
ui/),  an  open  source  API  that  enables  users  to  create  multi-touch  applications  easily  on  a  variety  of 
hardware  platforms.  Sparsh  UI  enabled  group-based  multi-touch  interaction  with  the  Virtual  Battlespace 
via  its  command  architecture. 

Physical  Vehicle  Integration  -  Cyberphysical  interfaces  to  the  Virtual  Battlespace  were  developed  by 
Professor  Somani’s  team  to  enable  interaction,  control,  and  telemetry  of  physical,  real-life  UAV/UGV 
platforms.  To  generalize  the  classes  of  air  and  ground  vehicles  that  Battlespace  could  integrate  and  control 
an  ad-hoc  onion  routing  architecture  via  a  TCP/IP  protocol  stack  was  implemented,  which  delivers 
Battlespace  data  packets  to  the  appropriate  application  thread  on  the  UAVs/UGVs  and  vice  versa.  This 
provided  four  benefits.  [I],  it  allowed  Battlespace  to  control  the  UAVs/UGVs  over  the  Internet,  hence 
isolating  the  operators  from  the  immediate  dangers  of  having  to  be  within  visual,  or  radio-range  to  the 
physical  mission.  [II],  due  to  the  distributed  nature  of  the  system,  there  is  no  single  link  that  can  be 
compromised  due  to  severing  the  connection  between  the  Battlespace  and  any  UAV/UGV.  [Ill],  the  design 
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prevents  intermediary  parties  from  knowing  the  origin,  destination,  or  contents  of  Battlespace 
communications  protocol,  thus  providing  forward  secrecy.  [IV],  it  was  particularly  well  suited  to  work  with 
the  existing  Virtual  Battlespace  command  architecture. 

Extension  into  LVC  -  Expertise  gained  through  the  course  of  the  project,  particularly  in  the  architecture  of 
state-of-the-art  DoD  simulation  technology,  as  well  as  in  the  variable  scale,  and  information  visualization 
techniques  developed,  enabled  to  the  same  core  team  (Gilbert,  Winer  and  Oliver)  to  develop  a  successful 
collaboration  with  the  US  Army  RDECOM  resulting  in  a  funded  project  called  “ Advanced  Live  Virtual  and 
Constructive  Training ”.  This  new  project,  which  commenced  on  10/01/09  and  is  funded  through  9/30/13 
for  a  total  of  $4.3M,  leverages  much  of  the  infrastructure  and  research  results  developed  during  the  course 
of  the  subject  UAV  project,  and  has  enabled  us  to  reach  a  much  broader  audience  of  DoD  stakeholders 
spanning  government  research  labs  and  the  DoD  contractor  community. 

First  Responder  Training  -  Professor  Keren’s  group  used  support  from  this  grant  to  explore  the 
feasibility  of  integrating  biometric  sensors  and  integrating  the  information  collected  to  enhance  decision¬ 
making  and  training.  The  result  is  a  new  research  platform  called  SEER:  Simulator  for  Enhancing 
Emergency  Response  Under  Stress.  Professor  Keren  and  his  team  work  closely  with  the  LVC  team  and  are 
currently  focused  on  securing  state-level  and  DHS  support  for  developing  first  responder  training.  A  white 
paper  describing  SEER  research  thrust  and  some  preliminary  results  are  attached  as  an  appendix  to  this 
report. 

Workshop/demo  -  As  the  Virtual  Teleoperation  for  UAVs  project  approached  its  conclusion  in  spring 
2011,  the  Pis  planned  a  workshop  to  feature  its  results  as  well  as  the  preliminary  results  and  vision  for  their 
recently  commenced  LVC  project.  The  workshop  at  Iowa  State  University’s  Virtual  Reality  Applications 
Center,  was  held  on  June  21,  and  brought  together  partners  from  The  Boeing  Company,  Rockwell  Collins, 
US  Army  RDECOM,  Wright-Patterson  AFB,  Dignitas  Technologies,  and  the  University  of  Iowa  to  explore 
opportunities  for  collaboration  in  next  generation  command  and  control  and  LVC  training.  Twenty-two 
attendees  gathered  to  view  demonstrations  of  each  other’s  work  and  hear  presentations  from  military 
stakeholders  about  current  DoD  research  needs.  The  VRAC  demo  featured  at  this  workshop  built 
significantly  on  the  Virtual  Battlespace  technologies,  integrating  them  with  systems  for  live,  virtual,  and 
constructive  training.  In  a  scenario  depicting  an  IED  diffusing  mission  students  played  the  role  of 
Command  Center  staff  using  the  Virtual  Battlespace  for  surveillance  with  UAVs  as  well  as  coordination  of 
a  UGV  for  disarming  the  IED.  The  overall  situational  awareness  afforded  by  the  Virtual  Battlespace  was 
integral  to  the  success  of  the  mission.  A  video  of  this  demo  can  be  seen  at: 
http://www.  youtube. com/watch?v=kWTvlxkziUk 
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